Empirically Explaining SGD from a Line Search Perspective

Optimization in Deep Learning is mainly guided by vague intuitions and strong assumptions, with a limited understanding how why these work practice. To shed more light on this, our provides some deeper understandings of SGD behaves empirically analyzing the trajectory taken from line search perspective. Specifically, costly quantitative analysis full-batch loss along trajectories common used models trained subset CIFAR-10 performed. Our core results include that lines update step direction highly parabolically. Further on, we show there exists learning rate which always performs almost exact searches loss. Finally, provide different perspective increasing batch size has same effect as decreasing factor.